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ABSTRACT 

There is a host of research examining the equivalence of alternative modes of technology-facilitated educational 
delivery (such as computer-based or online instruction] and traditional classroom instruction. While various studies have 
promoted each of these modalities for specific populations or topic areas, the bulk of research supports relative 
equivalence between student learning as a result of any of these forms of instruction. Unfortunately the majority of 
studies discuss ''learning" as an outcome variable with little consideration to relevant components of learning such as 
depth of knowledge or level of understanding. The purpose of fhis study was to examine the educational impact (depth 
of knowledge] of different forms of instruction (computer-based, lecture, and readings] aimed at teaching basic, 
introductory-level concepts. Eighty-eight participants received informational material via computer, lecture, or a 
written article. Participants' knowledge of this material was assessed using three types of questions (multiple choice, fill- 
in-the-blank, and essay] that examined three levels of understanding (rote, application, and evaluation] of material from 
an introductory-level course. The results indicated that computer-based instruction and readings produced a more 
evaluative understanding of the material than did lecture regardless of question type, but failed to detect significant 
differences in rote or application level understanding. In addition, computer-based instruction and readings were 
significantly more efficient means of instruction than lecture. The implication of these findings for educators is discussed. 


INTRODUCTION 

As universities move forward to keep up with rapidly 
advancing educational technology, there is a growing 
emphasis to incorporate computers and technology- 
based multimedia into the classroom. The increasing 
reliance on technology is rapidly changing the face of 
higher education and, in response to the growing 
demand, many publishing companies now offer optional, 
supplementary computer-based resources to 
complement designated textbooks. This leaves individual 
instructors with the responsibility of determining which 
resources are most effective for meeting their instructional 
goals. To complicate the issue further, there is a plethora of 
research literature examining the equivalence of 
traditional face-to-face and computer-assisted 
educational delivery modes, but the available research is 
often contradictory, inconclusive, or targeting only a 
narrow population or target area (for a comprehensive 
overview of relevant research, see Russell, 2005). In 
addition, the existing literature typically discusses student 


learning as a generalized outcome with no attention to 
differences in the type of knowledge or depth of 
understanding. The purpose of this study was to examine 
the educational impact (depth of knowledge) of different 
forms of instruction (computer-based, lecture, and 
readings) aimed at teaching basic, introductory-level 
concepts. 

Research Supporting Computer-Based Instruction 

Several meta-analyses (Cohen & Dacanay, 1994; Fletcher- 
Flinn & Gravatt, 1995; Kulik, 1994; Kulik & Kulik, 1991; Kulik, 
Kulik & Shwalb, 1986 and Shachar & Neumann, 2003) 
summarize the results of hundreds of studies that 
compared the effectiveness of computer-based 
instruction to face-to-face instruction and found enhanced 
student learning in response to technology-mediated 
instruction. Kulik (1994) identified five major points 
emerging from classroom research on computer-based 
instruction: 

• Learning outcomes are higher in classes that 
incorporate computer-based instruction than in 
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classes that rely solely on traditional instructional 
methods 

• Computer-based instruction is more efficient than 
traditional instruction 

• Attitudes toward instruction are enhanced after 
receiving computer-based instruction 

• Attitudes towards computer use are enhanced after 
receiving computer-based instruction 

• Computer-based instruction does not enhance 
attitudes toward subject matter 

Research Supporting Traditional Instruction 

In direct contrast to these reports, other research (Brown & 
Liedholm, 2002; Efendioglo & Murry, 2000; and 
Hartzoulakis, 2002) has found superior learning gains for 
students participating in a traditional face-to-face 
classroom experience. In all these studies, students who 
were exposed to familiar, traditional lecture settings 
(including video-simulated lectures, see Efendioglo & 
Murray, 2000) scored significantly higher on outcome 
learning measures than students who completed learning 
activities in an isolated, technologically-mediated setting 
(including virtual classes and computer-based instructional 
simulations). 

Research Supporting No Significant Differences in 
Instructional Mode 

When examining comparative research in the educational 
equivalence of various modes of instruction, the largest 
body of research reports no significant differences in 
student learning as a result of traditional or computer- 
based instructional strategies. As indicated by Russell 
(2001), over 350 published studies indicate that student 
learning is not differentially impacted as a result of either 
classroom instruction or computer-mediated, distance 
learning technologies. Russell's summaries are further 
supported by several meta-analyses (Cohen, Ebeling & 
Kulik, 1981; Liao, 1992; and Machtmes, 2000) that 
statistically analyzed the range of comparative studies 
available; the results of these meta-analyses show no 
evidence to indicate that computer-assisted instruction is 
any more or less effective than traditional face-to-face 
instruction. 


Summary 

As indicated by the vast range of contradictory conclusions 
concerning the general effectiveness of computer-based 
instruction, it is difficulty (if not impossible) to make 
generalized statements about the value of technology- 
mediated instruction as an overall tool to promote student 
learning. The term computer-based instruction 
encompasses a wide variety of instructional uses, including 
drill-and-practice, tutorial, dialogue, management, 
simulations, enrichment, programming, logo, and 
interactive applications. These diverse uses, as well as rapid 
advances in programming and applications, eliminate 
broad conclusions about the effectiveness of computer- 
based instruction. As Kulik (1994) effectively highlighted, 
"we need to go beyond generic conclusions and make 
statements about the effectiveness of specific types of 
computer-based instruction" (p. 22). 

Accordingly, the present study aimed to go beyond a 
general examination of student learning to look explicitly at 
the depth or type of learning gains that may be available 
via computer-based instruction. Specifically, the study 
compared the effectiveness of a typical computer-based 
instructional module, a traditional lecture and a written 
document on the level or depth of student understanding. 
Theoretically, if active learning results in better 
understanding, then the interactive components of 
computer-based instruction should enhance higher order 
thinking about the material. As such, outcome measures 
examined whether students who completed the 
computer-based instruction demonstrate a deeper or 
more evaluative level of knowledge than students who 
learned via traditional lecture. Another possible advantage 
to computer-based instruction is instructional efficiency 
(Kulik, 1994), so additional measures examined if students 
who learned via computer spent less time with the 
instructional material than students who listened to the 
lecture or completed the readings. 

Method 

Participants 

Eighty-eight participants were recruited from the 
Introductory Psychology subject pool at a large university in 
the Midwest for participation in this study; participants 
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received course credit in exchange for their participation. 
There were a mean of 29 participants in each of the three 
instructional conditions (computer-based, lecture, and 
readings); there were no between-group differences in 
gender, year in school, form of typical instruction, mean 
age (19.46 years), mean GPA (3.34), or pretest knowledge 
of topic. 

Independent Variables 
Methods of Instruction 

Three methods of instruction (computer-based module, 
videotaped lecture, and readings) were utilized to teach 
general learning principles from a standard introductory 
psychology text. The content of the instruction was held 
constant in all three instructional units. A commercially 
available, interactive CD-ROM (offered as a textbook 
supplement by the publisher) provided the computer- 
based instruction (Coon, 1998). The computer-based unit 
incorporated interactive activities, video/sound clips, and 
multiple-choice review questions in addition to a text 
description of the concepts. The videotaped lecture was 
developed to match the instructional content of the 
computerized unit and was delivered by a professional 
actor with experience as a college psychology teacher. 
The readings consisted of text description of the concepts 
accompanied by illustrations. All activities and videos 
found in the computer-based unit were described in the 
videotaped lecture and reading versions of instruction, in 
addition, participants in the videotaped lecture and 
readings condition were given review quizzes to match the 
multiple-choice review questions found in the 
computerized unit. 

Dependent Variables 
Measures of Learning 

A post-test measure was designed to assess various levels 
of understanding (rote, application, and evaluative) using 
various question formats (multiple-choice, fill-in-the-blank, 
and essay). All questions reflected typical knowledge- 
based assessment items based on a standard taxonomy 
of learning objectives (Bloom, Engelhart, Furst, Hill, & 
Kratwohl, 1956) that targeted various level of 
understanding 1 . 

Comment (DB1): 

Terrific idea; got to see the table to be sure it is a good sample 


Scoring was completed by a knowledgeable outside 
investigator 2 who was unfamiliar with the research 
hypotheses and blind to participants' instructional method. 
In order to maintain consistent scoring, a detailed scoring 
guide was utilized. In addition, a random sample of 80 
items was scored by a second skilled grader to assess the 
quality of the learning measure. The percentage of 
agreement between graders was above 85%, providing 
evidence of reliable scoring. 

Design and Analysis 

The design of the study was a between-groups comparison 
of instructional methods. The learning in each group was 
compared using an analysis of variance with the post-test 
assessment as the dependent variable. The post-test 
assessment was broken down by question type (multiple- 
choice, fill-in-the-blank, and essay) and by level of 
understanding (rote, application, and evaluative) for 
additional analyses. 

Procedure 

Each participant completed a pre-test of content 
knowledge and was randomly assigned to an instructional 
condition. All instruction was delivered individually in private 
rooms. Participants were informed that they had unlimited 
time to review and master the material. The post-test was 
administered immediately following the instructional unit 3 . 

Results 

An examination of the means of each instructional 
condition revealed a consistent pattern of findings with 
students learning via reading showing the greatest level of 
understanding, followed closely by students learning from 
computer-based instruction. As shown in Figure 1, students 
learning from either of these instructional methods showed 
increased understanding over students receiving 
information via lecture. This pattern of findings consistently 
emerged in several different partitions of the total data 
from the learning assessment. A series of between-groups 

Comment (DB2): "Investigator" may not provide the information needed; 
why would one think this person is a reasonable judge? Any TA experience? 
Maybe you don't want to identify as undergraduate, but relevant experience 
grading other materials would be helpful hint that scoring was competent 
(beyond the reliability data). 

Comment (DB3): Here again there is the issue of expectations; unlimited 
time within what frame of credit? Did people really have unlimited time or 
perceive that they did? You need to clarify this throughout the paragraph 
(that is, once for the whole paragraph). 
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Figure 1. Mean total test score by instructional mode. 

ANOVAs (.05 alpha level) were conducted to provide 
support for these findings. Means and standard deviations 
for each instructional condition are listed in Table 1, 2 lists 
the ANOVA results for all statistical analyses. 

Students learning via reading demonstrated a more 
evaluative understanding than students learning via 
computer or videotaped lecture (see Figure 2 ).Abetween- 
groups ANOVA on the level of understanding revealed a 
significant difference in students evaluative understanding 
(F(2,85) = 5.612, p = .005, MSE = 6.76)\ 


Means and standard deviations for each instructional condition 



Computer 

Mean SD 

Lecture 

Mean 

SD 

Readings 

Mean 

SD 

Pretest: 

8,41 

3.92 

8.84 

4.14 

9.31 

2.65 

Multiple-choice 

4.04 

2.28 

4.24 

2.23 

4.52 

1.88 

Essay 

4.38 

2.28 

4.60 

2.53 

4.79 

1.66 

Posttest: 

27.84 

10.18 

23.71 

7.81 

28.82 

8.16 

Rote: 

9.57 

4.22 

8.59 

3.51 

9.63 

3.45 

Multiple-choice 

2.61 

.83 

2.52 

.78 

2.74 

.58 

Fill-in-the-blank 

2.07 

.90 

1.97 

1.02 

2.26 

.82 

Essay 

4.89 

3.12 

4.10 

2.47 

4.63 

2.69 

Application: 

9.82 

3.86 

8.22 

3.12 

10.13 

3.18 

Multiple-choice 

2.14 

.65 

2.21 

.68 

2.39 

.72 

Fill-in-the-blank 

2.14 

.85 

1.97 

.68 

2.19 

.75 

Essay 

5.54 

3.19 

4.05 

2.74 

5.55 

2.58 

Evaluative: 

8.41 

2.98 

6.86 

2.43 

9.06 

2.38 

Multiple-choice 

1.07 

.72 

.66 

.72 

1.26 

.89 

Fill-in-the-blank 

2.64 

.56 

2.55 

.57 

2.55 

.57 

Essay 

4.73 

2.55 

3.66 

1.88 

5.26 

2.08 

Total multiple-choice 

5.82 

1.42 

5.38 

1.42 

6.39 

1.26 

Total fill-in-the-blank 

6.86 

1.78 

6.52 

1.53 

7.00 

1.67 

Total essay 

15.16 

8.25 

11.81 

6.01 

15.44 

6.56 

Instructional time: 

22.75 

7.53 

29.31 

3.32 

21.87 

7.38 

Test time: 

24.68 

7.24 

21.11 

7.03 

24.59 

8.01 


Table 1. Means and standard deviations for each instructional 
condition 


Comment (DB1): Throughout the results I would like each sentence 
to start with a prose description of the qualitative result; this first 
sentence only states that there was a difference and gives a lot of 
numbers, none of which are the means or other indication of 
difference. You need to make an argument with the figures 
showing that consistent pattern. I may need to see this in paper or 
with the figures embedded into the word document so I know 
what we have here. This long paragraph seems sterile to me 
without particulars. 


Complete ANOVA results 



F 

df 

P 

MSE 

Pretest 

.46 

2, 85 

.636 

13.01 

Multiple-choice 

.38 

2, 85 

.686 

4.54 

Essay 

.27 

2, 85 

.765 

4.73 

Posttest 

2.84 

2, 85 

.064 

76.50 

Rote 

.72 

2, 85 

.489 

13.94 

Multiple-choice 

.71 

2, 85 

.493 

.54 

Fill-in-the-blank 

.79 

2, 85 

.455 

.83 

Essay 

.61 

2, 85 

.549 

7.66 

Application 

2.68 

2, 85 

.075 

11.50 

Multiple-choice 

1.03 

2, 85 

.362 

.47 

Fill-in-the-blank 

.73 

2, 85 

.484 

.58 

Essay 

2.68 

2, 85 

.074 

8.06 

Evaluative 

5.61 

2, 85 

.005 

6.76 

Multiple-choice 

4.59 

2, 85 

.013 

.62 

Fill-in-the-blank 

.26 

2, 85 

.774 

.32 

Essay 

4.18 

2, 85 

.019 

4.75 

Total multiple-choice 

4.12 

2, 85 

.020 

1.86 

Total fill-in-the-blank 

.66 

2, 85 

.519 

2.76 

Total essay 

2.45 

2, 85 

.092 

48.70 

Instructional time 

11.87 

2, 85 

<.001 

40.86 

Testing time 

2.06 

2, 79 

.134 

55.56 


Table 2. Complete ANOVA results 


A Tukey HSD pairwise comparison showed that the 
evaluative knowledge of students who learned the material 
via readings (M = 9.06, SD = 2.38) was significantly higher 
than students receiving lecture instruction (M = 6.86, SD = 
2.43); but there was no significant difference in evaluative 
knowledge between students learning via computer (M = 
8.41, SD = 2.98) and the other two instructional conditions. 
In order to further examine differences in evaluative 
understandings, between-group ANOVAs were conducted 
on the type of question used to measure evaluative 
knowledge. A significant difference was found for both the 
multiple-choice (F (2, 85) = 4.59, p = .013, MSE = .62) and 
the essay (F (2,85) = 4.18, p = .019, MSE = 4.75) questions. 
Post hoc Tukey HSD comparisons revealed that for both the 
multiple-choice and evaluative questions, students who 
read the material (multiple-choice M = 1.26, SD = .89; 
essay M = 5.26, SD = 2.08) demonstrated a more 
evaluative understanding than students learning via 
lecture (multiple-choice M = ,66,SD = .72; essayM = 3.66, 
SD = 1.88). Again, there was no significant difference in 
evaluative knowledge as demonstrated in multiple-choice 
or essay questions between students receiving computer- 
based instruction (multiple-choice M = 1.07, SD = .72; 
essay M = 4.73, SD = 2.55) and the other two conditions 
(see Figure 3). 

Another between-groups ANOVA on question type 
revealed a significant difference in students' responses to 
multiple-choice questions (F (2,85) = 4.12, p = .02, MSE = 
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Figure 3. Mean test score by instructional mode and question type. 

1.86). A Tukey HSD pairwise comparison showed that 
students who read the material scored significantly better 
on multiple-choice questions (M = 6.39, SD = 1.26) than 
students learning via lecture (M = 5.38, SD = 1.42). As 
reported above, a breakdown of multiple-choice 
questions by level of understanding revealed a significant 
difference for evaluative multiple-choice questions. As 
shown in Table 2, there were no other significant findings for 
post-test comparisons. 

In an analysis of instructional time, a between-groups 
ANOVA showed a significant difference in the time students 
spent with the instructional material (F (2, 85) = 11.87, p < 
.001, MSE = 40.86). Post hoc Tukey's HSD comparisons 
revealed that students who learned via computer (M = 
22.75 minutes, SD = 7.53) or readings (M = 21.87 minutes, 
SD = 7.38) spent significantly less time with the instructional 
material than students who listened to the lecture (M = 
29.31 minutes, SD = 3.32); there was no significant 
difference in instructional time between students learning 
via computer or readings. To further examine the influence 
of instructional time, a Pearson's correlation analysis was 
conducted between instructional time and total post-test 
score; this analysis revealed no significant correlation (r(88) 
= .069, p = .524). While there was no significant difference 
in testing time (F (2,79) = 2.06, p = .134, MSE = 55.56), 
there was a significant positive correlation between testing 
time and total post-test score(r (82) = .472, p < .01) 
indicating that increased time spent taking the test led to 
higher post-test scores. 

Discussion 5 

Despite the large body of evidence indicating no 

Comment (DB5): I need to see the new results section and the 
figures before I can really comment on the discussion. 


significant differences between computer-based and 
traditional classroom instruction, it was hypothesized that 
students who learned via computer would demonstrate 
increased depth of understanding over students who 
listened to a lecture due to the interactive nature of 
computer-based instruction. While this pattern was 
repeatedly found throughout the results, the trend was not 
significant. The current research found that readings alone 
produced the greatest amount of learning followed closely 
by the computer-based instruction; both of these 
instructional methods produced considerably more 
student learning than listening to a traditional lecture. 

Despite findings that support the readings as a slightly more 
effective instructional mode than computer-based 
instruction, instructors should be cautious about substituting 
readings in place of alternative technology supplements 
as there may be differences in students' enthusiasm toward 
the instructional mode. The learning outcomes clearly 
support reading as an effective instructional tool, but 
promoting students' active engagement with written 
material is a challenge. Students often report very little 
interest in reading assignments and may fail to complete 
assigned readings due to this lack of interest. No qualitative 
data was collected in the current study to examine 
students' attitudes or enthusiasm toward various 
instructional modes, but research (see meta-analysis by 
Kulik & Kulik, 1991; Fletcher-Flinn & Gravatt, 1995) indicates 
that many students find computer-based instruction more 
interesting and engaging than other instructional methods. 
It is possible that computer-based instruction does not 
provide any advanced instructional components, but 
simply presents the material in a more interesting manner. 
From this pragmatic perspective, the key is not whether 
learning outcomes are higher from readings or computer- 
based instruction but which instructional mode will promote 
the greatest student involvement. Thus, regardless of 
learning outcomes, there may be value in incorporating 
computer-based instruction over simple reading, as 
students may be more likely to actively engage in the 
material. 

The hypothesis that students who learned via computer- 
based instruction would demonstrate a deeper or more 
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evaluative level of knowledge than students who learned 
via traditional lecture was also not fully supported. Students 
learning via computer did show a trend toward a more 
evaluative understanding of the material, but this trend did 
not reach significance. Surprisingly, students who read the 
material demonstrated the deepest or most evaluative 
understanding of the material. In addition, no significant 
differences were found between the instructional modes 
on either rote or application levels of understanding. 

Unlike a lecture, computer-based instruction gives student 
the opportunity to learn material in a nonlinear fashion. 
Students are able to review, skip ahead, repeat sections, 
and practice skills; these instructional opportunities may 
provide a basis for which students are able to monitor their 
own learning and adjust the instructional material to meet 
their progress. An unexpected discovery was the high level 
of learning gained from simply reading the material. At first 
glance, this finding seems somewhat counterintuitive due 
to the assumption that students need assistance (lecture) in 
order to learn novel material. But two components of the 
reading experience in this study make it unique: 1) students 
had no distractions during the readings (conditions which 
are quite rare for the typical student), and 2) students were 
given review questions at several points within the reading 
(review questions that they needed to answer before 
moving on). In addition, reading is a familiar instructional 
form to students. It is possible that in the future, as students 
become more familiar with the nonlinear instructional 
structure of computer-based instruction, they will have the 
knowledge and expertise to take full advantage of the 
instructional capabilities of a computer. But currently, 
reading (linear instruction) is a much more comfortable 
and familiar instructional strategy; so, in retrospect, it is not 
completely surprising that reading produced greater levels 
of understanding. 

A major difference between computer-based instruction 
and lecture or readings is the opportunity for the student to 
actively engage with the instructional material (through 
interactive or exploratory exercises). It is possible that a 
standard paper-and-pencil assessment (as used in this 
study) may not be adequate to detect potential 
differences in learning. Specifically, the one-on-one 


interaction available in computer-based instruction may 
enhance application or skill knowledge that is better 
demonstrated through skill-based measures. The 
interactivity provided by computer-based instruction may 
increase skills that are not measurable through standard 
assessment (see Mason & Bernstein, 2001). 

As hypothesized, computer-based instruction was more 
efficient than lecture in that students spent less time with the 
computer-based instructional unit than they did listening to 
the lecture, in addition, students learned as much, if not 
more, in less time. This has important implications for 
instructors, as they may be able to cover more material in 
less time providing increased time for other educational 
opportunities. It was interesting to note that while the lecture 
condition required significantly more instructional time than 
the other two instructional conditions, students who learned 
via lecture spent considerably less time testing. While it can 
be argued that the apparent results are more a function of 
testing time than instructional mode, students learning via 
computer and readings were apparently more engaged 
in the material as evidenced in the increased time spent 
testing. These findings effectively echo previous research 
that reports increased instructional efficiency (Kulik & Kulik, 
1991; Cohen & Decaney, 1994; Fletcher-Flinn & Gravatt, 
1995) for computer-based instruction than for traditional 
instruction, with 91 % of 32 studies finding that computer- 
based instruction was more efficient (taking only 2/3 as 
much time) as traditional instruction. 

These findings have important implications for instructors 
looking for instructional strategies to maximize student 
learning. While computer-based instructional supplements 
may not produce superior learning gains compared to 
other forms of instruction, the novelty of this instructional 
delivery method may foster increased student interest in 
course material over more traditional instructional 
strategies; this. Thus, it appears that promoting active 
engagement with the material, whether this occurs via 
targeted readings, computer-based instruction or other 
innovative instructional methods, is essential for student 
mastery of new information. In addition, computer-based 
supplements may encourage students to prepare more 
thoroughly for class by providing an efficient means of 
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preparation, thus allowing valuable class time to be spent 
on more advanced activities. 
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